Lawmakers that wish for Veterans Affairs Secretary Eric Shinseki to go are growing in number. President Obama may meet with him as early as Friday.